25/02/2024 - 02/03/2024

26/02/2024 06:51

I was able to get the readstatus.py script to "work":

[root@dhcp-10-163-105-238 software]# python3 read_status.py 1 11

Crate 01, Slot 11
Traceback (most recent call last):
  File "read_status.py", line 55, in <module>
    wfd.dispatch()
uhal._core.UdpTimeout: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.11:50001
[root@dhcp-10-163-105-238 software]# python3 read_status.py 1 11

Crate 01, Slot 11
Traceback (most recent call last):
  File "read_status.py", line 55, in <module>
    wfd.dispatch()
uhal._core.UdpTimeout: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.11:50001

I also tried the "working" IP:

[root@dhcp-10-163-105-238 software]# python3 read_status.py 1 41

Crate 01, Slot 41
Traceback (most recent call last):
  File "read_status.py", line 55, in <module>
    wfd.dispatch()
uhal._core.UdpTimeout: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.41:50001

[root@dhcp-10-163-105-238 software]#
[root@dhcp-10-163-105-238 software]# python3 read_status.py 1 41

Crate 01, Slot 41
Traceback (most recent call last):
  File "read_status.py", line 55, in <module>
    wfd.dispatch()
uhal._core.UdpTimeout: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.41:50001

[root@dhcp-10-163-105-238 software]#

Unsurprisingly we have a communicatation problem right now.

What I did to get the wfdConfig scripts installed:

  1. This is built upon my environment installed by my installer (https://github.com/PIONEER-Experiment/gm2daq-installer)
  2. cd /home/installation_testing/packages/experiment/gm2daq/environment_setup (replace with the correct path to environment_setup)
  3. ./detect_environment.sh (populations environment_variables.txt with locations of dependencies, you only need to do this once)
  4. cd ../../.. (returns to packages directory, but the software can be installed anywhere)
  5. git clone git@github.com:PIONEER-Experiment/wfdConfig.git
  6. export ADDRESS_TABLES=/home/installation_testing/packages/wfdConfig/software/address_tables/ (replace this with correct path to address_tables directory)
  7. python3 read_status.py 1 11

26/02/2024 07:14

I checked the BIOS on the 'be' computer and was unable to find any setting that corresponded to IPMI communcation.

26/02/2024 07:41

I noticed the version of our Nat-MCH is rather old from the sticker left on the back
fd8b5869180e72457b43ee198a8c79d2.png
Here is a guide that is closer to the correct version (but still not the correct version!):
https://www.slac.stanford.edu/grp/ssrl/spear/epics/vme/NAT_MCH-UsersManual_V122.pdf

26/02/2024 07:55

I'm able to use ipmitool to communicate with the MCH to retrieve some basic information:

[root@dhcp-10-163-105-238 software]# ipmitool -H 192.168.1.41 -P "" fru print
FRU Device Description : Builtin FRU Device (ID 0)
 Invalid FRU size 0

FRU Device Description : ICL-CERN FC7 (ID 13)
 Unsupported device

FRU Device Description : VT VT095 (ID 41)
 Unsupported device

FRU Device Description : VT VT095 (ID 40)
 Unsupported device

FRU Device Description : CU WFD5 (ID 15)
 Unsupported device

FRU Device Description : BU AMC13 (ID 30)
 Unsupported device

FRU Device Description : NAT-MCH-MCMC (ID 3)
 Unsupported device

FRU Device Description : VT UTC010 (ID 50)
 Unsupported device

FRU Device Description : NMCH-ShM
 Device not present (Destination unavailable)
[root@dhcp-10-163-105-238 software]# ipmitool -H 192.168.1.41 -P "" fru print
FRU Device Description : Builtin FRU Device (ID 0)
 Invalid FRU size 0

FRU Device Description : ICL-CERN FC7 (ID 13)
 Unsupported device

FRU Device Description : VT VT095 (ID 41)
 Unsupported device

FRU Device Description : VT VT095 (ID 40)
 Unsupported device

FRU Device Description : CU WFD5 (ID 15)
 Unsupported device

FRU Device Description : BU AMC13 (ID 30)
 Unsupported device

FRU Device Description : NAT-MCH-MCMC (ID 3)
 Unsupported device

FRU Device Description : VT UTC010 (ID 50)
 Unsupported device

FRU Device Description : NMCH-ShM
 Device not present (Destination unavailable)

This is nothing more than what we got from telnet, but it's proof that there is nothing "blocking" ipmitool from functioning.

26/02/2024 08:17

This first command:

[root@dhcp-10-163-105-238 lxedaq]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 1 -t 0x82 raw 0x32 0x34 1 11 0 4
Unable to send RAW command (channel=0x1 netfn=0x32 lun=0x0 cmd=0x34)
[root@dhcp-10-163-105-238 lxedaq]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 1 -t 0x82 raw 0x32 0x34 1 11 0 4
Unable to send RAW command (channel=0x1 netfn=0x32 lun=0x0 cmd=0x34)

reaches an error almost immediately (order of ms)

This second command

[root@dhcp-10-163-105-238 lxedaq]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0x82 raw 0x32 0x34 1 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)
[root@dhcp-10-163-105-238 lxedaq]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0x82 raw 0x32 0x34 1 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)

Takes longer to reach the same errors (~1 second). This makes me think this command is reaching some sort of timeout

This -b parameter is the transit channel, which I guess makes sense that we need to pick the right one. The same thing happens if you independently vary the -T parameter. I think these two parameters are specify the "out" route in the MCH, so when you get that correct it has some timeout while it waits for a response.

The -t parameter is like the destination, while everything after raw are commands/command parameters.

26/02/2024 09:20

Copying commands from the MCH manual gives this error:

[root@dhcp-10-163-105-238 amc13Config]# ipmitool –H 192.168.1.41 –P "" –T 0x82 –B 0 –t 0xc2 –b 7 hpm upgrade
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
[root@dhcp-10-163-105-238 amc13Config]# ipmitool –H 192.168.1.41 –P "" –T 0x82 –B 0 –t 0xc2 –b 7 hpm upgrade
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

This is a very stupid error that's hidden to the user. In short, the hyphens are different ASCII characters. ipmitool gets confused by the copy pasted version, but not the typed version. Example:

[root@dhcp-10-163-105-238 amc13Config]# ipmitool –H 192.168.1.41 –P "" –T 0x82 -b 7 –t 0x82 hpm upgstatus
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0x82 hpm upgstatus

PICMG HPM.1 Upgrade Agent 1.0.9:

^C
[root@dhcp-10-163-105-238 amc13Config]# ipmitool –H 192.168.1.41 –P "" –T 0x82 -b 7 –t 0x82 hpm upgstatus
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0x82 hpm upgstatus

PICMG HPM.1 Upgrade Agent 1.0.9:

^C

26/02/2024 09:37

There's an example command in the manual (https://www.slac.stanford.edu/grp/ssrl/spear/epics/vme/NAT_MCH-UsersManual_V122.pdf) to upgrade the firmware.

Interestingly, I get different errors depending on which module I try it on.

First I tried the power module 1

[root@dhcp-10-163-105-238 amc13Config]#  ipmitool -I lan -H 192.168.1.41 -A none -T 0x82 -B 0 -t 0xc2 -b 7 hpm upgrade f
w.hpm

PICMG HPM.1 Upgrade Agent 1.0.9:

Error getting device ID.
Verify whether the Target board is present

[root@dhcp-10-163-105-238 amc13Config]#
[root@dhcp-10-163-105-238 amc13Config]#  ipmitool -I lan -H 192.168.1.41 -A none -T 0x82 -B 0 -t 0xc2 -b 7 hpm upgrade f
w.hpm

PICMG HPM.1 Upgrade Agent 1.0.9:

Error getting device ID.
Verify whether the Target board is present

[root@dhcp-10-163-105-238 amc13Config]#

for sanity, I tried power module 2 as well, though I can visually see our power module is plugged into the 'PM1' slot

[root@dhcp-10-163-105-238 amc13Config]#  ipmitool -I lan -H 192.168.1.41 -A none -T 0x82 -B 0 -t 0xc4 -b 7 hpm upgrade f
w.hpm

PICMG HPM.1 Upgrade Agent 1.0.9:

Error getting device ID.
Verify whether the Target board is present
[root@dhcp-10-163-105-238 amc13Config]#  ipmitool -I lan -H 192.168.1.41 -A none -T 0x82 -B 0 -t 0xc4 -b 7 hpm upgrade f
w.hpm

PICMG HPM.1 Upgrade Agent 1.0.9:

Error getting device ID.
Verify whether the Target board is present

I also tried these commands to see if there was any variance in error messages depending on what parameters I use:

[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0xc2 hpm upgstatus

PICMG HPM.1 Upgrade Agent 1.0.9:

Error getting upgrade status. Failed to get response.
[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0xc2 hpm upgstatus

PICMG HPM.1 Upgrade Agent 1.0.9:

Error getting upgrade status. Failed to get response.
[root@dhcp-10-163-105-238 amc13Config]# ipmitool -I lan -H 192.168.1.41 -A none -T 0x82 -b 7 -t 0xc2 hpm upgstatus

PICMG HPM.1 Upgrade Agent 1.0.9:

Error getting upgrade status. Failed to get response.
[root@dhcp-10-163-105-238 amc13Config]# ipmitool -I lan -H 192.168.1.41 -A none -T 0x82 -b 7 -t 0xc2 hpm upgstatus

PICMG HPM.1 Upgrade Agent 1.0.9:

Error getting upgrade status. Failed to get response.
[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0xc2 hpm upgrade fw.hpm

PICMG HPM.1 Upgrade Agent 1.0.9:

Error getting device ID.
Verify whether the Target board is present
[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0xc2 hpm upgrade fw.hpm

PICMG HPM.1 Upgrade Agent 1.0.9:

Error getting device ID.
Verify whether the Target board is present

This leads me to believe there is no effective difference between using the -P flag vs the -I and -A flags.

26/02/2024 09:55

With the USB to microUSB plugged into the MCH, I get this error spam

[root@dhcp-10-163-105-238 amc13Config]# telnet 192.168.1.41
Trying 192.168.1.41...
Connected to 192.168.1.41.
Escape character is '^]'.


Welcome to NAT-MCH

nat> WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000
WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000
WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000
WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000
WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000 
[root@dhcp-10-163-105-238 amc13Config]# telnet 192.168.1.41
Trying 192.168.1.41...
Connected to 192.168.1.41.
Escape character is '^]'.


Welcome to NAT-MCH

nat> WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000
WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000
WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000
WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000
WARNING: usb: TX buffer(8) pending, dtdt=0x00088080, EPSR=0x000a0004, EPCR1=0x00880000

unplugging the USB and power cycling the crate seemed to fix the issue.

26/02/2024 10:09

To view the MCH webserver on a computer with a GUI you can just ssh port forward the webpage server:

ssh -L 8080:192.168.1.41:80 root@10.163.105.238
ssh -L 8080:192.168.1.41:80 root@10.163.105.238

This maps the MCH webserver to your computer's localhost:8080, so then just enter localhost:8080 into your webrowser to see the webpage. Credentials are
Username: root
Password: nat

26/02/2024 11:39

NatView software description link: https://nateurope.com/product/natview-mtca-management/
Free version download link (requires account): https://nateurope.com/dl.php?file=2592

I'm skeptical this will help us because the description just says "On request NATview can scan a selectable range of IP addresses and will display those assigned to a NAT-MCH" which to me seems functionally the same as my brute force IP checker.

It would also require a computer with a GUI (it's probably possible but very nontrivial to forward the GUI output of a java application).

27/02/2024 16:16

The output in terminal and insidetelnet 192.168.1.41 when trying to run the readIPs.py -s 13 script:

[root@dhcp-10-163-105-238 amc13Config]# ./readIPs.py -s 13
Reading IP addresses of board in slot 13 from host 192.168.1.41
T2 IP Bytes:
ipmitool -H 192.168.1.41 -U '' -P '' -T 0x82 -b 7 -t 0xa4 raw 0x32 0x34 0 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)
[]
[]
T1 IP Bytes:
ipmitool -H 192.168.1.41 -U '' -P '' -T 0x82 -b 7 -t 0xa4 raw 0x32 0x34 1 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)
[]
[]
[root@dhcp-10-163-105-238 amc13Config]# ./readIPs.py -s 13
Reading IP addresses of board in slot 13 from host 192.168.1.41
T2 IP Bytes:
ipmitool -H 192.168.1.41 -U '' -P '' -T 0x82 -b 7 -t 0xa4 raw 0x32 0x34 0 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)
[]
[]
T1 IP Bytes:
ipmitool -H 192.168.1.41 -U '' -P '' -T 0x82 -b 7 -t 0xa4 raw 0x32 0x34 1 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)
[]
[]
SESSION(0): activated session 0x00000005
R(30,16,1)R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)R(30,16,2)WARN - SMQ(0x81->0x20->0x82:0): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:1): GET_ADDRESS_INFO_REQ - timeout remove
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:3): GET_ADDRESS_INFO_REQ - timeout remove
R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
WARN - SMQ(0x81->0x20->0x82:4): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:5): GET_ADDRESS_INFO_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:1): GET_ADDRESS_INFO_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
WARN - SMQ(0x81->0x20->0x82:2): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:6): NETFN_CONTR1_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)R(30,16,2)WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 0
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 1
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
WARN - SMQ(0x81->0x20->0x82:0): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:1): NETFN_CONTR1_REQ - timeout remove
WARN - SESSION(0): discarded 1 pending messages
SESSION(0): closing session 0x00000005
SESSION(0): activated session 0x00000005
R(30,16,1)R(30,16,2)WARN - SMQ(0x10->0x82->0xa4:3): NETFN_CONTR1_REQ - timeout remove
ERR - SESSION: cannot get SEND_MESSAGE_REQ from queue
ERR - RMCP: could not find nat0 session for msg
ERR - CSIF: xfer failed - SHM(20)->SMS(81) IPMI msg seq 0x20 SEND_MESSAGE_RSP
ERR - LSHM(0): failed - SHM(20)->SMS(81) IPMI msg seq 0x20 SEND_MESSAGE_RSP
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:5): NETFN_CONTR1_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 2
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 3
R(30,16,2)WARN - SMQ(0x81->0x20->0x82:0): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:1): GET_ADDRESS_INFO_REQ - timeout remove
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:3): GET_ADDRESS_INFO_REQ - timeout remove
R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
WARN - SMQ(0x81->0x20->0x82:4): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:5): GET_ADDRESS_INFO_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 0
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 1
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:1): GET_ADDRESS_INFO_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
WARN - SMQ(0x81->0x20->0x82:2): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:6): NETFN_CONTR1_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)R(30,16,2)WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 2
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 3
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
WARN - SMQ(0x81->0x20->0x82:0): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:1): NETFN_CONTR1_REQ - timeout remove
WARN - SESSION(0): discarded 1 pending messages
SESSION(0): closing session 0x00000005
R(30,16,1)R(30,16,2)WARN - SMQ(0x10->0x82->0xa4:3): NETFN_CONTR1_REQ - timeout remove
ERR - SESSION: cannot get SEND_MESSAGE_REQ from queue
ERR - RMCP: could not find nat0 session for msg
ERR - CSIF: xfer failed - SHM(20)->SMS(81) IPMI msg seq 0x31 SEND_MESSAGE_RSP
ERR - LSHM(0): failed - SHM(20)->SMS(81) IPMI msg seq 0x31 SEND_MESSAGE_RSP
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:5): NETFN_CONTR1_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 0
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 1
R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 2
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 3
R(30,16,1)R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
R(30,16,1)R(30,16,2)WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 0
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 1
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
R(30,16,1)R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
SESSION(0): activated session 0x00000005
R(30,16,1)R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)R(30,16,2)WARN - SMQ(0x81->0x20->0x82:0): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:1): GET_ADDRESS_INFO_REQ - timeout remove
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:3): GET_ADDRESS_INFO_REQ - timeout remove
R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
WARN - SMQ(0x81->0x20->0x82:4): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:5): GET_ADDRESS_INFO_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:1): GET_ADDRESS_INFO_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
WARN - SMQ(0x81->0x20->0x82:2): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:6): NETFN_CONTR1_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)R(30,16,2)WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 0
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 1
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
WARN - SMQ(0x81->0x20->0x82:0): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:1): NETFN_CONTR1_REQ - timeout remove
WARN - SESSION(0): discarded 1 pending messages
SESSION(0): closing session 0x00000005
SESSION(0): activated session 0x00000005
R(30,16,1)R(30,16,2)WARN - SMQ(0x10->0x82->0xa4:3): NETFN_CONTR1_REQ - timeout remove
ERR - SESSION: cannot get SEND_MESSAGE_REQ from queue
ERR - RMCP: could not find nat0 session for msg
ERR - CSIF: xfer failed - SHM(20)->SMS(81) IPMI msg seq 0x20 SEND_MESSAGE_RSP
ERR - LSHM(0): failed - SHM(20)->SMS(81) IPMI msg seq 0x20 SEND_MESSAGE_RSP
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:5): NETFN_CONTR1_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 2
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 3
R(30,16,2)WARN - SMQ(0x81->0x20->0x82:0): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:1): GET_ADDRESS_INFO_REQ - timeout remove
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:3): GET_ADDRESS_INFO_REQ - timeout remove
R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
WARN - SMQ(0x81->0x20->0x82:4): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:5): GET_ADDRESS_INFO_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 0
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 1
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:1): GET_ADDRESS_INFO_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
WARN - SMQ(0x81->0x20->0x82:2): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:6): NETFN_CONTR1_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)R(30,16,2)WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 2
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 3
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
WARN - SMQ(0x81->0x20->0x82:0): SEND_MESSAGE_REQ - timeout remove
WARN - SMQ(0x10->0x82->0xa4:1): NETFN_CONTR1_REQ - timeout remove
WARN - SESSION(0): discarded 1 pending messages
SESSION(0): closing session 0x00000005
R(30,16,1)R(30,16,2)WARN - SMQ(0x10->0x82->0xa4:3): NETFN_CONTR1_REQ - timeout remove
ERR - SESSION: cannot get SEND_MESSAGE_REQ from queue
ERR - RMCP: could not find nat0 session for msg
ERR - CSIF: xfer failed - SHM(20)->SMS(81) IPMI msg seq 0x31 SEND_MESSAGE_RSP
ERR - LSHM(0): failed - SHM(20)->SMS(81) IPMI msg seq 0x31 SEND_MESSAGE_RSP
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)WARN - SMQ(0x10->0x82->0xa4:5): NETFN_CONTR1_REQ - timeout remove
ERR - SMQ(0x82->0x10): SEND_MESSAGE_RSP - unexpected response
ERR - SEND_MSG_RSP(0x82->0x10): no outstanding message
WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 0
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 1
R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=9 GET_ADDRESS_INFO_REQ
R(30,16,1)R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 2
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 3
R(30,16,1)R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
R(30,16,1)R(30,16,2)WARN - FMQ(0x20->0x82->0xa8): GET_FAN_LEVEL_REQ - timeout remove idx 0
WARN - FMQ(0x20->0x82->0xaa): GET_FAN_LEVEL_REQ - timeout remove idx 1
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
R(30,16,1)R(30,16,2)ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message
ERR - FMQ(0xa8->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xa8->0x82): no outstanding message
ERR - FMQ(0xaa->0x82): GET_FAN_LEVEL_RSP - unexpected response
ERR - FMSG_RSP(0xaa->0x82): no outstanding message

A lot of timeouts.

In contrast, this is what happens when I intentionall pick a wrong target address for the carrier:

[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x84 -b 7 -t 0xa4 raw 0x32 0x34 0 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)
[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x84 -b 7 -t 0xa4 raw 0x32 0x34 0 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)
SESSION(0): activated session 0x00000005
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x09 SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x09 SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x09 SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x09 SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x0a SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x0a SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x0a SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x0a SEND_MESSAGE_REQ
SESSION(0): closing session 0x00000005
SESSION(0): activated session 0x00000005
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x09 SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x09 SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x09 SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x09 SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x0a SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x0a SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x0a SEND_MESSAGE_REQ
WARN - CMU(0): invalid i2c dst - input SHM(10)->CM(84) BRIDGED IPMI msg seq 0x0a SEND_MESSAGE_REQ
SESSION(0): closing session 0x00000005

If I turn on IPMI debug on, I can see the request being transferred, but nothing recieved.

Command:

[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0xa4 raw 0x32 0x34 0 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)
[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0xa4 raw 0x32 0x34 0 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)

telnet output:

TX CM->MMC: *** GET_ADDRESS_INFO_REQ ***
  src: rqAddr 0x10 LUN 2   dest:  rsAddr 0xa4 LUN 0   rqSeq: 0x09
  CRC1: 0xac (ok)   CRC2: 0xc9 (ok)
  message data (len 1):
  00
TX CM->MMC: *** GET_ADDRESS_INFO_REQ ***
  src: rqAddr 0x10 LUN 2   dest:  rsAddr 0xa4 LUN 0   rqSeq: 0x09
  CRC1: 0xac (ok)   CRC2: 0xc9 (ok)
  message data (len 1):
  00

What's curious to me is it looks like the message data is not what we're trying to send.

If I reboot the module and telnet in, this are all the messages I can see as it boots up:

[root@dhcp-10-163-105-238 ~]# telnet 192.168.1.41
Trying 192.168.1.41...
Connected to 192.168.1.41.
Escape character is '^]'.


Welcome to NAT-MCH

nat> ..
PM(50): State change M1->M2
PM(50): State change M2->M3
PM(50): State change M3->M4 (global status 0x06 ok - Primary)
PM(50): FRU active, state M4
PM(50): changed to NORMAL mode
LSHM(0): CM sensor 68 LUN 0 <unknown> hotswap M1->M2
LSHM(0): FRU 50 sensor 37 LUN 0 'PM HOT SWAP' hotswap M2->M3
LSHM(0): FRU 50 sensor 37 LUN 0 'PM HOT SWAP' hotswap M3->M4
PM1(50): PM module at ic2=0xc2 chosen as the primary PM !
mcmc(3): M0->M1
mcmc(3): reading 512 bytes of fru data
Base AVR version is 1.2
...........PM1(50): determining type of Module in MCH 2 slot - AMC
CU1(40): started management task for i2c=0xa8
CU1(40): State change M0->M1
CU2(41): started management task for i2c=0xaa
CU2(41): State change M0->M1
.
mcmc(3): M1->M2
mcmc(3): M2->M3
PM1: payload already ON for FRU 3
WARN - LSHM(0): ignore version change sensor
LSHM(0): CM sensor 108 LUN 0 <unknown> hotswap M1->M2
LSHM(0): FRU 3 sensor 114 LUN 0 'HotSwap' hotswap M2->M3
CU1(40): reading 1024 bytes of fru data
CU2(41): reading 1024 bytes of fru data
mcmc(3): M3->M4
mcmc(3): M4 (MCH FRU active)
AMC26(30): M0->M1
AMC9(13): M0->M1
AMC11(15): M0->M1
LSHM(0): FRU 3 sensor 114 LUN 0 'HotSwap' hotswap M3->M4
PM1 Event(50): new power state=0x5b for channel 1
PM1 Event(50): new power state=0x1b for channel 2
PM1 Event(50): new power state=0x1b for channel 3
PM1 Event(50): new power state=0x1b for channel 4
PM1 Event(50): new power state=0x1b for channel 13
PM1 Event(50): new power state=0x1b for channel 15
LSHM(0): FRU 50 sensor 34 LUN 0 'FET TEMP' temperature 'upper non-recoverable go high'
LSHM(0): FRU 50 sensor 34 LUN 0 'FET TEMP' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 34 LUN 0 'FET TEMP' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 33 LUN 0 'BRICK 1 TEMP' temperature 'upper non-recoverable go high'
LSHM(0): FRU 50 sensor 33 LUN 0 'BRICK 1 TEMP' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 33 LUN 0 'BRICK 1 TEMP' temperature 'upper non-critical go high'
AMC26(30): reading 152 bytes of fru data
AMC11(15): reading 232 bytes of fru data
AMC9(13): reading 512 bytes of fru data
..LSHM(0): FRU 50 sensor 32 LUN 0 'BRICK 2 TEMP' temperature 'upper non-recoverable go high'
LSHM(0): FRU 50 sensor 32 LUN 0 'BRICK 2 TEMP' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 32 LUN 0 'BRICK 2 TEMP' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 31 LUN 0 'UTC010 tIN' temperature 'upper non-recoverable go high'
LSHM(0): FRU 50 sensor 31 LUN 0 'UTC010 tIN' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 31 LUN 0 'UTC010 tIN' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 30 LUN 0 'UTC010 tOUT' temperature 'upper non-recoverable go high'
LSHM(0): FRU 50 sensor 30 LUN 0 'UTC010 tOUT' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 30 LUN 0 'UTC010 tOUT' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 29 LUN 0 'UTC010 Brick T1' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 29 LUN 0 'UTC010 Brick T1' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 28 LUN 0 'UTC010 Brick T2' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 28 LUN 0 'UTC010 Brick T2' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 27 LUN 0 'UTC010 Brick T3' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 27 LUN 0 'UTC010 Brick T3' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 26 LUN 0 'UTC010 Brick T4' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 26 LUN 0 'UTC010 Brick T4' temperature 'upper non-critical go high'
...R(40,13,1)...bp_getCurrent(13): no Current Descriptor found for channel !
bp_getCurrent(15): no Current Descriptor found for channel !
.........
AMC11(15): Handle=0x01 - closed
AMC11(15): M1->M2
.....LSHM(0): CM sensor 118 LUN 0 <unknown> hotswap M1->M2
...............
.AMC26(30): Handle=0x01 - closed
AMC26(30): M1->M2
......LSHM(0): CM sensor 129 LUN 0 <unknown> hotswap M1->M2
.......
AMC9(13): Handle=0x01 - closed
AMC9(13): M1->M2
LSHM(0): CM sensor 138 LUN 0 <unknown> hotswap M1->M2
R(41,13,1)CU1(40): Cooling Unit needs 5.0 Amps power
CU2(41): Cooling Unit needs 5.0 Amps power
CU1(40): Cooling unit has 20 sensors, reading ...
CU2(41): Cooling unit has 17 sensors, reading ...
......................................
CU2(41): State change M1->M2
CU2(41): State change M2->M3
PM1: payload already ON for FRU 41
...
LSHM(0): CM sensor 169 LUN 0 <unknown> hotswap M1->M2
LSHM(0): FRU 41 sensor 208 LUN 0 'CU HOT SWAP' hotswap M2->M3
CU1(40): State change M1->M2
CU1(40): State change M2->M3
PM1: payload already ON for FRU 40
Activation: modules are ready
Activation: all modules ready, Allowance Period (30 sec) stopped - continue with module startup !
CU2(41): State change M3->M4
CU2(41): FRU active (state M4)
CU2(41)  fan speed properties:
  minimum speed level:    0x05
  maximum speed level:    0x3f
  normal operating level: 0x20
  fan tray properties:    0x00
LSHM(0): FRU 40 sensor 189 LUN 0 'CU HOT SWAP' hotswap M1->M2
LSHM(0): CU2 FRU 41 added
LSHM(0): FRU 40 sensor 189 LUN 0 'CU HOT SWAP' hotswap M2->M3
LSHM(0): FRU 41 sensor 208 LUN 0 'CU HOT SWAP' hotswap M3->M4
LSHM(0): CU2  properties min 5 max 63 norm 32
LSHM(0): CU2  now operational at level 22 (29%)
AMC26(30): M2->M3
PM1: payload already ON for FRU 30
CU1(40): State change M3->M4
CU1(40): FRU active (state M4)
CU1(40)  fan speed properties:
  minimum speed level:    0x05
  maximum speed level:    0x3f
  normal operating level: 0x20
  fan tray properties:    0x00
LSHM(0): FRU 30 sensor 136 LUN 0 'Hotswap' hotswap M2->M3
LSHM(0): CU1 FRU 40 added
LSHM(0): FRU 40 sensor 189 LUN 0 'CU HOT SWAP' hotswap M3->M4
LSHM(0): CU1  properties min 5 max 63 norm 32
LSHM(0): CU1  now operational at level 22 (29%)
AMC26(30): M3->M4
AMC26(30): State M4 (FRU active)
LSHM(0): FRU 30 sensor 136 LUN 0 'Hotswap' hotswap M3->M4
AMC9(13): M2->M3
PM1: payload already ON for FRU 13
LSHM(0): FRU 13 sensor 167 LUN 0 'FC7 HS' hotswap M2->M3
AMC11(15): M2->M3
PM1: payload already ON for FRU 15
AMC9(13): M3->M4
AMC9(13): State M4 (FRU active)
LSHM(0): FRU 15 sensor 127 LUN 0 'Hotswap' hotswap M2->M3
LSHM(0): FRU 13 sensor 167 LUN 0 'FC7 HS' hotswap M3->M4
AMC11(15): M3->M4
AMC11(15): State M4 (FRU active)
LSHM(0): FRU 15 sensor 127 LUN 0 'Hotswap' hotswap M3->M4
[root@dhcp-10-163-105-238 ~]# telnet 192.168.1.41
Trying 192.168.1.41...
Connected to 192.168.1.41.
Escape character is '^]'.


Welcome to NAT-MCH

nat> ..
PM(50): State change M1->M2
PM(50): State change M2->M3
PM(50): State change M3->M4 (global status 0x06 ok - Primary)
PM(50): FRU active, state M4
PM(50): changed to NORMAL mode
LSHM(0): CM sensor 68 LUN 0 <unknown> hotswap M1->M2
LSHM(0): FRU 50 sensor 37 LUN 0 'PM HOT SWAP' hotswap M2->M3
LSHM(0): FRU 50 sensor 37 LUN 0 'PM HOT SWAP' hotswap M3->M4
PM1(50): PM module at ic2=0xc2 chosen as the primary PM !
mcmc(3): M0->M1
mcmc(3): reading 512 bytes of fru data
Base AVR version is 1.2
...........PM1(50): determining type of Module in MCH 2 slot - AMC
CU1(40): started management task for i2c=0xa8
CU1(40): State change M0->M1
CU2(41): started management task for i2c=0xaa
CU2(41): State change M0->M1
.
mcmc(3): M1->M2
mcmc(3): M2->M3
PM1: payload already ON for FRU 3
WARN - LSHM(0): ignore version change sensor
LSHM(0): CM sensor 108 LUN 0 <unknown> hotswap M1->M2
LSHM(0): FRU 3 sensor 114 LUN 0 'HotSwap' hotswap M2->M3
CU1(40): reading 1024 bytes of fru data
CU2(41): reading 1024 bytes of fru data
mcmc(3): M3->M4
mcmc(3): M4 (MCH FRU active)
AMC26(30): M0->M1
AMC9(13): M0->M1
AMC11(15): M0->M1
LSHM(0): FRU 3 sensor 114 LUN 0 'HotSwap' hotswap M3->M4
PM1 Event(50): new power state=0x5b for channel 1
PM1 Event(50): new power state=0x1b for channel 2
PM1 Event(50): new power state=0x1b for channel 3
PM1 Event(50): new power state=0x1b for channel 4
PM1 Event(50): new power state=0x1b for channel 13
PM1 Event(50): new power state=0x1b for channel 15
LSHM(0): FRU 50 sensor 34 LUN 0 'FET TEMP' temperature 'upper non-recoverable go high'
LSHM(0): FRU 50 sensor 34 LUN 0 'FET TEMP' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 34 LUN 0 'FET TEMP' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 33 LUN 0 'BRICK 1 TEMP' temperature 'upper non-recoverable go high'
LSHM(0): FRU 50 sensor 33 LUN 0 'BRICK 1 TEMP' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 33 LUN 0 'BRICK 1 TEMP' temperature 'upper non-critical go high'
AMC26(30): reading 152 bytes of fru data
AMC11(15): reading 232 bytes of fru data
AMC9(13): reading 512 bytes of fru data
..LSHM(0): FRU 50 sensor 32 LUN 0 'BRICK 2 TEMP' temperature 'upper non-recoverable go high'
LSHM(0): FRU 50 sensor 32 LUN 0 'BRICK 2 TEMP' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 32 LUN 0 'BRICK 2 TEMP' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 31 LUN 0 'UTC010 tIN' temperature 'upper non-recoverable go high'
LSHM(0): FRU 50 sensor 31 LUN 0 'UTC010 tIN' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 31 LUN 0 'UTC010 tIN' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 30 LUN 0 'UTC010 tOUT' temperature 'upper non-recoverable go high'
LSHM(0): FRU 50 sensor 30 LUN 0 'UTC010 tOUT' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 30 LUN 0 'UTC010 tOUT' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 29 LUN 0 'UTC010 Brick T1' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 29 LUN 0 'UTC010 Brick T1' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 28 LUN 0 'UTC010 Brick T2' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 28 LUN 0 'UTC010 Brick T2' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 27 LUN 0 'UTC010 Brick T3' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 27 LUN 0 'UTC010 Brick T3' temperature 'upper non-critical go high'
LSHM(0): FRU 50 sensor 26 LUN 0 'UTC010 Brick T4' temperature 'upper critical go high'
LSHM(0): FRU 50 sensor 26 LUN 0 'UTC010 Brick T4' temperature 'upper non-critical go high'
...R(40,13,1)...bp_getCurrent(13): no Current Descriptor found for channel !
bp_getCurrent(15): no Current Descriptor found for channel !
.........
AMC11(15): Handle=0x01 - closed
AMC11(15): M1->M2
.....LSHM(0): CM sensor 118 LUN 0 <unknown> hotswap M1->M2
...............
.AMC26(30): Handle=0x01 - closed
AMC26(30): M1->M2
......LSHM(0): CM sensor 129 LUN 0 <unknown> hotswap M1->M2
.......
AMC9(13): Handle=0x01 - closed
AMC9(13): M1->M2
LSHM(0): CM sensor 138 LUN 0 <unknown> hotswap M1->M2
R(41,13,1)CU1(40): Cooling Unit needs 5.0 Amps power
CU2(41): Cooling Unit needs 5.0 Amps power
CU1(40): Cooling unit has 20 sensors, reading ...
CU2(41): Cooling unit has 17 sensors, reading ...
......................................
CU2(41): State change M1->M2
CU2(41): State change M2->M3
PM1: payload already ON for FRU 41
...
LSHM(0): CM sensor 169 LUN 0 <unknown> hotswap M1->M2
LSHM(0): FRU 41 sensor 208 LUN 0 'CU HOT SWAP' hotswap M2->M3
CU1(40): State change M1->M2
CU1(40): State change M2->M3
PM1: payload already ON for FRU 40
Activation: modules are ready
Activation: all modules ready, Allowance Period (30 sec) stopped - continue with module startup !
CU2(41): State change M3->M4
CU2(41): FRU active (state M4)
CU2(41)  fan speed properties:
  minimum speed level:    0x05
  maximum speed level:    0x3f
  normal operating level: 0x20
  fan tray properties:    0x00
LSHM(0): FRU 40 sensor 189 LUN 0 'CU HOT SWAP' hotswap M1->M2
LSHM(0): CU2 FRU 41 added
LSHM(0): FRU 40 sensor 189 LUN 0 'CU HOT SWAP' hotswap M2->M3
LSHM(0): FRU 41 sensor 208 LUN 0 'CU HOT SWAP' hotswap M3->M4
LSHM(0): CU2  properties min 5 max 63 norm 32
LSHM(0): CU2  now operational at level 22 (29%)
AMC26(30): M2->M3
PM1: payload already ON for FRU 30
CU1(40): State change M3->M4
CU1(40): FRU active (state M4)
CU1(40)  fan speed properties:
  minimum speed level:    0x05
  maximum speed level:    0x3f
  normal operating level: 0x20
  fan tray properties:    0x00
LSHM(0): FRU 30 sensor 136 LUN 0 'Hotswap' hotswap M2->M3
LSHM(0): CU1 FRU 40 added
LSHM(0): FRU 40 sensor 189 LUN 0 'CU HOT SWAP' hotswap M3->M4
LSHM(0): CU1  properties min 5 max 63 norm 32
LSHM(0): CU1  now operational at level 22 (29%)
AMC26(30): M3->M4
AMC26(30): State M4 (FRU active)
LSHM(0): FRU 30 sensor 136 LUN 0 'Hotswap' hotswap M3->M4
AMC9(13): M2->M3
PM1: payload already ON for FRU 13
LSHM(0): FRU 13 sensor 167 LUN 0 'FC7 HS' hotswap M2->M3
AMC11(15): M2->M3
PM1: payload already ON for FRU 15
AMC9(13): M3->M4
AMC9(13): State M4 (FRU active)
LSHM(0): FRU 15 sensor 127 LUN 0 'Hotswap' hotswap M2->M3
LSHM(0): FRU 13 sensor 167 LUN 0 'FC7 HS' hotswap M3->M4
AMC11(15): M3->M4
AMC11(15): State M4 (FRU active)
LSHM(0): FRU 15 sensor 127 LUN 0 'Hotswap' hotswap M3->M4

29/02/2024 09:37

I just noticed that the command

[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0xa4 raw 0x32 0x34 0 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)
[root@dhcp-10-163-105-238 amc13Config]# ipmitool -H 192.168.1.41 -P "" -T 0x82 -b 7 -t 0xa4 raw 0x32 0x34 0 11 0 4
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x34)

telnet output:

TX CM->MMC: *** GET_ADDRESS_INFO_REQ ***
  src: rqAddr 0x10 LUN 2   dest:  rsAddr 0xa4 LUN 0   rqSeq: 0x09
  CRC1: 0xac (ok)   CRC2: 0xc9 (ok)
  message data (len 1):
  00
TX CM->MMC: *** GET_ADDRESS_INFO_REQ ***
  src: rqAddr 0x10 LUN 2   dest:  rsAddr 0xa4 LUN 0   rqSeq: 0x09
  CRC1: 0xac (ok)   CRC2: 0xc9 (ok)
  message data (len 1):
  00

also has:

TX CM->MMC: *** NETFN_CONTR1_REQ ***
  src: rqAddr 0x10 LUN 2   dest:  rsAddr 0xa4 LUN 0   rqSeq: 0x0a
  CRC1: 0x94 (ok)   CRC2: 0x83 (ok)
  message data (len 4):
  00 0b 00 04
TX CM->MMC: *** NETFN_CONTR1_REQ ***
  src: rqAddr 0x10 LUN 2   dest:  rsAddr 0xa4 LUN 0   rqSeq: 0x0a
  CRC1: 0x94 (ok)   CRC2: 0x83 (ok)
  message data (len 4):
  00 0b 00 04

The message data here corresponds to the command parameters:
0 11 0 4
In hex

I think the "real" error is that the amc13 is not responding:

ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ
ipmi_SendFru(30): timeout - no response for REQ: 0x10->0xa4, Seq=10 NETFN_CONTR1_REQ

29/02/2024 12:08

From /.../wfdConfig/software/read_addresses.py I editted the script to print out the IPMI command used. IT looked liek this:

[root@dhcp-10-163-105-238 software]# python3 read_addresses.py 1 1

Crate   Slot   S/N   IP Address        MAC Address
ipmitool -I lan -H 192.168.1.15 -U shelf -P shelf -B 0x0 -T 0x82 -b 7 -t 0x72 raw 0x32 0x53
ipmitool -I lan -H 192.168.1.15 -U shelf -P shelf -B 0x0 -T 0x82 -b 7 -t 0x72 raw 0x32 0x52
[root@dhcp-10-163-105-238 software]# python3 read_addresses.py 1 1

Crate   Slot   S/N   IP Address        MAC Address
ipmitool -I lan -H 192.168.1.15 -U shelf -P shelf -B 0x0 -T 0x82 -b 7 -t 0x72 raw 0x32 0x53
ipmitool -I lan -H 192.168.1.15 -U shelf -P shelf -B 0x0 -T 0x82 -b 7 -t 0x72 raw 0x32 0x52

I changed the fields for username, password, IP, and the target address -t to reflect our setup:
ipmitool -I lan -H 192.168.1.41 -P "" -B 0x0 -T 0x82 -b 7 -t 0x86 raw 0x32 0x53

I get all the same issues as when trying to communicate with the AMC

[root@dhcp-10-163-105-238 software]# ipmitool -I lan -H 192.168.1.41 -P "" -B 0x0 -T 0x82 -b 7 -t 0x86 raw 0x32 0x53
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x53)
[root@dhcp-10-163-105-238 software]# ipmitool -I lan -H 192.168.1.41 -P "" -B 0x0 -T 0x82 -b 7 -t 0x86 raw 0x32 0x53
Unable to send RAW command (channel=0x7 netfn=0x32 lun=0x0 cmd=0x53)
TX CM->MMC: *** GET_ADDRESS_INFO_REQ ***
  src: rqAddr 0x10 LUN 2   dest:  rsAddr 0x86 LUN 0   rqSeq: 0x09
  CRC1: 0xca (ok)   CRC2: 0xc9 (ok)
  message data (len 1):
  00
TX CM->MMC: *** GET_ADDRESS_INFO_REQ ***
  src: rqAddr 0x10 LUN 2   dest:  rsAddr 0x86 LUN 0   rqSeq: 0x09
  CRC1: 0xca (ok)   CRC2: 0xc9 (ok)
  message data (len 1):
  00
TX CM->MMC: *** NETFN_CONTR1_REQ ***
  src: rqAddr 0x10 LUN 2   dest:  rsAddr 0x86 LUN 0   rqSeq: 0x0a
  CRC1: 0xb2 (ok)   CRC2: 0x73 (ok)
  no message data
TX CM->MMC: *** NETFN_CONTR1_REQ ***
  src: rqAddr 0x10 LUN 2   dest:  rsAddr 0x86 LUN 0   rqSeq: 0x0a
  CRC1: 0xb2 (ok)   CRC2: 0x73 (ok)
  no message data
ipmi_SendFru(15): timeout - no response for REQ: 0x10->0x86, Seq=9 GET_ADDRESS_INFO_REQ
ipmi_SendFru(15): timeout - no response for REQ: 0x10->0x86, Seq=9 GET_ADDRESS_INFO_REQ

29/02/2024 12:21

ChatGPT's explanation of all the debug information:

TX CM->MMC: This indicates that the message is being transmitted from the Command Manager to the Management Controller.

NETFN_CONTR1_REQ: This is the type of IPMI message being sent. It's a request related to some control function (contr1). NETFN stands for Network Function, which specifies the type of operation being requested.

src: This section provides information about the source of the message.

rqAddr 0x10: This is the request address, indicating the source address of the message. In this case, it's 0x10.
LUN 2: This indicates the Logical Unit Number associated with the source of the message. LUN values specify the logical unit within a device that the message is intended for.
dest: This section provides information about the destination of the message.

rsAddr 0x86: This is the response address, indicating the destination address of the message. In this case, it's 0x86.
LUN 0: This indicates the Logical Unit Number associated with the destination of the message.
rqSeq: This is the request sequence number, indicating the sequence number associated with the request. In this case, it's 0x0a.

CRC1 and CRC2: These are cyclic redundancy check values used to ensure the integrity of the message data during transmission. CRC1 and CRC2 are computed checksums that are compared against expected values to verify that the message hasn't been corrupted.

no message data: This indicates that there is no message data included with this particular message. It's just a control request without additional payload data.